This is exactly the same as the others but we’re going to limit ourselves only to the people in Condition 3 of Experiment 1 who we could fit well, just to see if our results are robust to that. If they change dramatically that indicates that they are being shaped by people we can’t fit anyway.

Let’s get rid of people

I’m going to arbitrarily decide to get rid of the people whose fit in the (beta,gamma) case is worse than 0.01 for either of the FIVE or UNLIMITED parts of the experiment. This removes 31 out of 107 people.

Conditions 1 and 2

First let’s have a look at what is called Condition 1 and Condition 2 in the paper, i.e., in Experiment 1 the people who only saw five datapoints (always four red and one blue) and the ones who saw as many as they wanted; I’ve called these FIVE and UNLIMITED here.

Just so we can compare them directly, let’s put both posteriors on top of each other. We can see that as we would expect people tighten their generalisations with additional data.

Out of curiosity, let’s also see how many items people selected in the unlimited condition.

Condition 3 (ALL)

Now let’s take a look at the condition where we got priors, estimates after five, and estimates after an unlimited number; this was called Condition 3 in the paper and I’m calling it ALL here. Our first question is whether making them do the prior estimate made them much different from when they didn’t. So we want to compare five to five and unlimited to unlimited.

Each of these graphs show the data for Condition 3 (ALL), with the analogous data from the other ones shown as a dashed line and Xes.

These are virtually identical to the previous conditions (I’m not sure that correlation makes sense here but the graphs are pretty much superimposed on each other so I’m not worried. If we wanted to I suppose we could do t-tests at each point but that’s a lot of tests and it seems unnecessary).

Anyway though let’s also just sanity check that the number selected isn’t different across the two unlimited conditions.

These aren’t very different which we can verify with a t-test.

t.test(selected ~ condition, data=de1un_comp)
## 
##  Welch Two Sample t-test
## 
## data:  selected by condition
## t = 1.7374, df = 139.64, p-value = 0.08452
## alternative hypothesis: true difference in means between group all and group unlimited is not equal to 0
## 95 percent confidence interval:
##  -0.5435495  8.4242104
## sample estimates:
##       mean in group all mean in group unlimited 
##                21.98684                18.04651

This all suggests that people didn’t do weird things when they were asked to make estimates three times. So now let’s look at the overall priors they reported!

Overall it is sensible - mostly uniform, slightly peaked at 50%, no a priori preference for red or blue. But of course what we’re interested in is individual variation, not the overall.

Looking at fitted values: \(\beta\), \(\delta\), and \(\gamma\)

Let’s have a look at the distribution of parameters in Experiment 1. As a reminder:

\(\beta\): determines to what degree participants use their stated prior, and to what degree they use the uniform distribution to deduce their posterior. If \(\beta=1\) that means it’s all prior, if \(\beta=0\) it is all uniform distribution

\(\gamma\): how much they weighted the red chips they saw. if \(\gamma=1\) it is veridical, lower is underweighting, higher is overweighting.

\(\delta\): how much they weighted the blue chips they saw. if \(\delta=1\) it is veridical, lower is underweighting, higher is overweighting.

And also a 3D plot

Finally, let’s look at histograms of all of the variables.

Looking at fitted values: beta alone

I wanted to see if the beta values would be different if we weren’t also fitting delta and gamma. So let’s look at that histogram.

For the priors: 34.6% in FIVE and 31.8% in UNLIMITED had \(\beta\) less than 0.1, and 8.4% in FIVE and 8.4% in UNLIMITED had \(\beta\) greater than 0.9.

Looking at fitted values: \(\gamma\) alone

I wanted to see if the \(\gamma\) values would be different if we weren’t also fitting \(\delta\) and \(\beta\) So let’s look at that histogram.

For the likelihoods: 26.2% in Fɪᴠᴇ and 36.4% in Uɴʟɪᴍɪᴛᴇᴅ had \(\gamma\) less than 1 (i.e., were conservative).

Looking at fitted values: \(\beta\) and \(\gamma\) alone

Here we assume one parameter (call it \(\gamma\)) instead of two separate ones (\(\gamma\) and \(\delta\)). i.e. this forces them to be the same and can be thought of as a conservatism parameter. I wanted to look at this because I think it might be a lot more interpretable than having both.

\(\beta\): determines to what degree participants use their stated prior, and to what degree they use the uniform distribution to deduce their posterior. If \(\beta=1\) that means it’s all prior, if \(\beta=0\) it is all uniform distribution

\(\gamma\): how much they weighted the chips they saw. if \(\gamma=1\) it is veridical, lower is underweighting, higher is overweighting.

Let’s calculate the Spearman correlation on the non-log values:

# five
cor.test(de1_fittwo$beta[de1_fittwo$section=="Fɪᴠᴇ"],
         de1_fittwo$gamma[de1_fittwo$section=="Fɪᴠᴇ"],
         method="spearman")
## Warning in cor.test.default(de1_fittwo$beta[de1_fittwo$section == "Fɪᴠᴇ"], :
## Cannot compute exact p-value with ties
## 
##  Spearman's rank correlation rho
## 
## data:  de1_fittwo$beta[de1_fittwo$section == "Fɪᴠᴇ"] and de1_fittwo$gamma[de1_fittwo$section == "Fɪᴠᴇ"]
## S = 75150, p-value = 0.8146
## alternative hypothesis: true rho is not equal to 0
## sample estimates:
##         rho 
## -0.02734786
# unlimited
cor.test(de1_fittwo$beta[de1_fittwo$section=="Uɴʟɪᴍɪᴛᴇᴅ"],
         de1_fittwo$gamma[de1_fittwo$section=="Uɴʟɪᴍɪᴛᴇᴅ"],
         method="spearman")
## Warning in cor.test.default(de1_fittwo$beta[de1_fittwo$section ==
## "Uɴʟɪᴍɪᴛᴇᴅ"], : Cannot compute exact p-value with ties
## 
##  Spearman's rank correlation rho
## 
## data:  de1_fittwo$beta[de1_fittwo$section == "Uɴʟɪᴍɪᴛᴇᴅ"] and de1_fittwo$gamma[de1_fittwo$section == "Uɴʟɪᴍɪᴛᴇᴅ"]
## S = 69277, p-value = 0.6497
## alternative hypothesis: true rho is not equal to 0
## sample estimates:
##        rho 
## 0.05294552

And a histogram of them too

For the priors: 43.4% in FIVE and 50% in UNLIMITED had \(\beta\) less than 0.1, and 18.4% in FIVE and 21.1% in UNLIMITED had \(\beta\) greater than 0.9.

For the likelihoods: 52.6% in FIVE and 57.9% in UNLIMITED had \(\gamma\) less than 1 (i.e., were conservative).

Individuals: After five items

So now let’s look at individuals - compare their prior and posterior after five, based on the best-fit \(\beta\) and \(\gamma\)

red line: their reported prior

green line: their reported posterior after five

dotted grey: Bayes rule prediction assuming uniform prior (not shown)

solid grey: Bayes rule prediction assuming their prior

dotted black: line based on best-fit \(\beta\) and \(\gamma\)

Let’s also look at this for the ones where we only fit \(\beta\).

And then with both \(\gamma\) and \(\delta\) as well as \(\beta\).

Individuals: After unlimited items

We’re going to do the same thing as before but this time comparing their prior and posterior after all of their items.

red line: their reported prior

blue line: their reported posterior after unlimited

dotted grey: Bayes rule prediction assuming uniform prior (not shown)

solid grey: Bayes rule prediction assuming their prior

dotted black: line based on best-fit \(\beta\) and \(\gamma\)

We can also get a sense of how good the fits were.

## TableGrob (1 x 2) "arrange": 2 grobs
##   z     cells    name           grob
## 1 1 (1-1,1-1) arrange gtable[layout]
## 2 2 (1-1,2-2) arrange gtable[layout]

Let’s also look at this for the ones where we only fit beta.

And then with both gamma and delta as well as beta.

Aggregate fits

We can also look at the parameter values for the aggregate fits. First, FIVE ITEMS

Then, UNLIMITED ITEMS